With this PR in place, in PY2 mode, the following example (simplified from some real-world code) gives an error:

def f(x):
    return x.strip()
g = lambda line: map(f, line.split(':'))
lines = ['']
dict([g(line) for line in lines if len(g(line)) == 2])  # <-- here

The error is:

__tmp__.py:5: error: List comprehension has incompatible type List[List[Any]]

I keep going back and forth over what's going on in that example. You'd think the type of g(line) should be List[str] because line is a str. Then the problem is that dict([['']]) is wrongly typed. (The code intends to make a list of pairs of strings, but it uses List[str] instead of Tuple[str, str], and then the argument to dict() is wrong.)

But the type of g(line) is actually List[Any] because it is equivalent to

def g(line: Any) -> XXX:
    return map(f, line.split(':'))

where XXX is the type of the return expression, i.e. map(f, line.split(':')) -- which is List[Any] because at this point line is only known to have the type Any. This is shown by checking

g = lambda line: [line]
reveal_type(g(''))  # E: Revealed type is 'builtins.list[Any]'

Without this PR the type would just be Any, shutting up the type checker.

So on the one hand I conclude that the original code (from which I boiled this down) would have a type error even if we had coerced the type of g to Callable[[str], List[str]]. So the PR actually found a type error that was hidden before because any lambda would be inferred as type Any without a type context.

OTOH maybe we could do better by constructing a generic function from the lambda, so that the types here would correspond:

g = lambda line: [line]  # corresponds to Callable[[T], List[T]]

Inferring generic lambda will be more precise, which will probably catch more errors; it won't silence anything that's not silenced now. So this suggestion is independent from the original example.

One potential problem with inferring generic lambda is that it might be too strict - it will be hard to specify Any for some arguments selectively. Why don't we infer generic function parameters?

Inferring generic types for lambdas (when there's no type context) would require some heavy machinery (unless we special case some simple cases only). We only infer lambda argument types in trivial cases where the type context has a callable type, and I don't think that anything substantially more complicated is worth the effort right now.

Instead of inferring Any as the argument type when there is no help from type context, it might be more consistent to give up and ask help from the user, similar to how we don't infer List[Any] for [] when we can't determine the item type. However, annotating lambdas using # type: Callable[<...>] is much less obvious than annotating lists, and lambdas without type context don't seem to be very common, so falling back to Any doesn't sound very bad. Maybe we should at least give an error if there is an implicit Any argument type when using --disallow-untyped-defs? We can create a follow-up issue for that.

Remember, the lambda syntax does not support type annotations.

Brainstorming a little more, maybe in

f = lambda x: ...
foo(f)

we could just save the whole lambda as the value/type of f, and type-check the use of f as if it was the original lambda? The places where I've seen code like this looked like the only reason to introduce the variable was to break a long line.

(I'm also okay with merging this and maybe iterating later.)

Hm, I found another regression in a different codebase.

import re
p = re.compile(r".*")
print(p.sub(lambda *a, **k: "ho", "abc"))

This was fine before, but now says

__tmp__.py:3: error: Cannot infer type of lambda

But the following fails with or without this PR:

from typing import Callable
def f(t: Callable[[str], str]) -> str: ''
f(lambda *_: "")  # E: Cannot infer type of lambda

(Since callable_ctx.arg_kinds != arg_kinds)

It feels like it is unrelated to this PR, since this error happens before the decision to infer "something". I will try to find the cause anyway; maybe I'm wrong.

(Sorry: of course this is related to this PR since there's no error without it)

Ok, I get it: the error "occurs" without this PR too, but returning AnyType() has the implicit meaning of silencing this error. We cannot infer the type of the lambda even though it has context, but we do not notify the user.

If so, I should move and rephrase the TODO.
(Also, I am not sure this specific case is an example of where we can't reasonably infer the type).

I think this specific case can be treated separately. For now there's a bandage on it.

(I am also not sure about my previous conclusion that Any silences the errors - it could be something to do with the overload in the example)

So I have one more real-world issue with this. Consider this example:

from typing import Callable
def f(g = lambda *a, **k: None):
    # type: (Callable[..., None]) -> None
    g(1, 2)

This gives an error:

__tmp__.py:2: error: Incompatible types in assignment (expression has type Callable[[StarArg(Any), KwArg(Any)], None], variable has type Callable[..., None])

The problem is that I don't even know how to silence this -- you can't put # type: ignore on a # type comment and I don't know how to make the arguments more general than .... The only solution would be to use plain Callable but I would prefer to keep the None return.

Thoughts?

Turns out it's not about the parameters but about the return type. When you say Callable[..., None] mypy understand Callable[..., Void] but the lambda was inferred to return builtins.None.

There's actually a test for this behavior, which fixed #1425.
Changing the inference to return Void does not resurface #1425. It does fail this test:

[case testLambdaReturningNone]
f = lambda: None
x = f()

Yielding error: Function does not return a value. I am not sure that this error is not needed - this code is obviously wrong, and in more complex situations I would expect the lambda to have context. (Thinking about it, the original problem should not have occurred at all, since it does have context; we shouldn't be inferring blindly in the first place).

In other words, for this PR to work we need to decide whether lambda: None is a procedure, or whether a function that returns builtins.None is compatible with Callable[..., Void].

I would expect Callable[..., T] to be compatible with Callable[..., Void] for all T - in a way, Void is the top element - but there are explicit checks to avoid None <: Void and UninhabitedType <: Void even though both seem fine to me. I am probably missing something, but if it's about catching erroneous returns even though they are technically sound, then IMHO these checks should not be part of the subtyping relation but an explicit special-casing somewhere else.

And maybe it's not worth the trouble.

Thanks for getting to the bottom of this. Your analysis sounds right, but I don't know what we should do about it either. Maybe @JukkaL has an opinion? I'll also ask @ddefisher.

We think this will also break some things in our codebases; I'll report back with results in a little while.

Actually it didn't break anything (and of course it fixed the example I gave). So I'm going to merge this.

There may be a follow-up issue -- do we still need the "Cannot infer type of lambda" error?

I have no idea.

Hm, actually the issue in #2634 (comment) is still there, so I filed #2764 for that.